With increase in camera speed and refinement of imaging elements, it has become difficult to ensure sufficient amounts of exposure at the time of image capturing. For this reason, captured images include a lot of noise. Here, the noise in these images can be removed by signal processing. This noise removal can be roughly classified into spatial filtering and temporal noise removal.
Noise removal using a spatial filter has problems that edges in an image are blurred in most cases and the saturation may decrease. On the other hand, temporal noise removal makes it possible to suppress the problems that occur in the noise removal using such a spatial filter and effectively removing noise.
Some exemplary conventional temporal noise removal schemes are intended to remove noise by adding plural images in the time direction. Some of the temporal noise removal schemes use motion compensation, and the others do not use motion compensation.
FIG. 11 is a functional block diagram showing a functional structure of a conventional image processing apparatus which performs a temporal noise removal scheme without using motion compensation.
As shown in FIG. 11, the image processing apparatus 70 includes a memory 71, an addition rate calculation unit 75, and a pixel addition unit 76.
First, the addition rate calculation unit 75 calculates addition rates using the input image and a reference image stored in the memory 71. Based on these addition rates, the pixel addition unit 76 adds the pixels of the input image and the pixels of the reference image.
Random noise in the input image is different from random noise in the reference image. For this reason, the random noise is statistically cancelled (the random noise is temporally smoothed) by adding the input image and the reference image. This reduces the random noise.
However, this scheme has a problem that, when there is a motion of a subject between the input image and the reference image, such pixel addition produces a residual image in the resulting output image.
FIG. 12 is a functional block diagram showing a functional structure of a conventional image processing apparatus which performs a temporal noise removal scheme using motion compensation.
As shown in FIG. 11, the image processing apparatus 70 includes a memory 71, a motion estimation unit 72, a motion compensation unit 73, an addition rate calculation unit 75, and a pixel addition unit 76.
First, the motion estimation unit 72 performs motion estimation using the input image and the reference image stored in the memory 71 to determine the positions of corresponding blocks in the reference image with respect to the respective blocks in the input image. Next, the motion compensation unit 73 generates block images of the reference image at the positions corresponding to the respective blocks in the input image according to the motion information estimated by the motion estimation unit 72. The addition rate calculation unit 75 calculates the addition rates using the input image and the reference image after motion compensation.
Next, the pixel addition unit 76 reduces, based on the addition rates, the noise by adding the pixels of the input image and the pixels of the reference image after motion compensation. In this way, the pixel addition unit 76 can reduce random noise in an image of a subject that makes the motion without producing any residual image in the resulting output image, by using the reference image after motion compensation.
However, there are cases where this scheme is not sufficient to perform accurate motion estimation due to influence of noise in images. In such a case, an image region at the non-correspondence positions is added, which causes an adverse effect that especially still regions look fluctuating in resulting output images.
Furthermore, some of temporal noise removal schemes combine a scheme without using motion compensation and a scheme using motion compensation (for example, see PTL (Patent Literature) 1).
FIG. 13 is a functional block diagram showing a functional structure of a conventional image processing apparatus which performs a temporal noise removal scheme that is a combination of a scheme without using motion compensation and a scheme using motion compensation.
As shown in FIG. 13, the image processing apparatus 70 includes a memory 71, a motion estimation unit 72, a motion compensation unit 73, a pixel addition unit 76, a selection control unit 77, and a reference image setting unit 78.
First, the motion estimation unit 72 and the motion compensation unit 73 perform motion estimation and motion compensation, respectively, as in the descriptions given with reference to FIG. 12. Next, the selection control unit 77 performs motion estimation using the input image and the reference image, and discriminates motion regions and motionless regions, based on the result of the estimation.
Next, the reference image setting unit 78 selects a reference image for each of the motion regions and motionless regions. More specifically, the reference image setting unit 78 selects, for each motion region, a reference image after motion compensation, and selects, for each motionless region, a reference image before motion compensation.
Next, the pixel addition unit 76 adds the pixels of the input image and the pixels of the reference image. In this way, random noise is temporally smoothed so as to reduce noise. Furthermore, selecting a reference image based on presence/absence of a motion in the input image reduces noise without producing a residual image even in a motion region.